SGL: optimize Qwen3.6 27B and 35B A3B online fp8 quantization performance on BMG w/o XPU Graph and resolve acc issue. - #577

…back

Add esimd_resadd_norm_gemv2_fp8_pert: fused residual-add + RMSNorm +
two fp8 per-tensor GEMVs, with the post-add residual (hidden+residual)
written to a separate new_residual buffer by the gid==0 group, removing
the follow-up aten::add. Used by the GDN and full-attention input-norm
fusions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- moe_up_merged_e5m2: single dispatch for routed + shared up projections.
- MoeRouterTopKNormFusedE4M3: fused post-attn norm + fp8 router + topk.
- moe_down_finalize_fused_e5m2: optional (env SGL_XPU_MOE_DOWN_FUSED,
  default off) folding of routed down into the finalize kernel.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Refresh patches/sglang_for_multi_arc.patch from origin/dev-bmg so the
Docker build picks up the fused-kernel Python changes (env-read caching,
MoE decode fusion, GDN/full-attention input-norm fusions).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…g chunks + lsc_load_2d)

Optimize the ESIMD fp8 MoE prefill up/down DPAS kernels for long prefill
(4k target). Cuts moe up+down from ~560ms to ~185ms (4096-token prefill,
Qwen3.5-35B-A3B, TP=2, xpu-graph disabled), all changes bit-exact.

- doubleGRF: add `-options -doubleGRF` to the moe_fp8 ext (setup.py). The
  kernel had a 192B register spill; doubleGRF eliminates it.

- Load-balancing chunks: add MpChunk{eid,t0,nt} + a single-WI
  moe_prefill_build_chunks_kernel that pre-splits each expert's token range
  into uniform MAX_M blocks. Change up/down grid from (num_experts, N-tile)
  with an inner m_base loop to (num_chunks, N-tile) with one block per WI,
  removing the serial tail that hot (heavily-routed) experts caused under
  skewed routing. Bit-exact: each output element is still summed over the
  full K by one WI in the same order.

- lsc_load_2d weight loading: replace the per-ns 8x tiny block_load loop
  with a single xesimd::lsc_load_2d 2D block load (N rows x 16 K, row-major)
  in both up and down kernels. Row-major result matches the old layout
  exactly (bit-exact).

- Merge build_chunks: factor up/down into *_core impls that accept
  prebuilt chunks; the fused moe_prefill_full_fp8 driver builds chunks once
  and shares them between up and down (valid since MP_UP_MAXM==MP_DOWN_MAXM),
  halving build_chunks launches (80->40 per prefill pass).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

… runtime

- patches/sglang_for_multi_arc.patch: regenerate from origin/dev-bmg
  (fp8 rowwise fused _scaled_mm + ESIMD fast-path wiring + GGUF q8_0 XPU
  methods). Verified applies cleanly to upstream v0.5.13.
- moe_prefill_fp8: MoE down kernel N tile 32->64 (issues N/32 lsc_load_2d
  blocks since 2D block height caps at 32 rows).
- docker: swap in oneCCL 2021.15 runtime (Arc Pro enhancements), replacing
  the torch-bundled libccl via /opt/venv/lib symlinks (Dockerfile +
  Dockerfile.dev).
- scripts: build_image tag -> intel registry 0728; add update_sglang_patch.sh
  to regenerate the multi-arc patch from a dev-bmg checkout.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ose copy)

Add moe_up_routed_e5m2_nmajor_kernel and moe_up_merged_e5m2_nmajor_kernel that
read sglang's native FusedMoE w13 layout [E, 2*inter, hidden] directly via
lsc_load_2d + fp8e5m2_block_to_vnni_nk, mirroring the existing e4m3 N-major
kernels. Flip the five e5m2 fused decode entries (moe_forward_full, _rtfused,
_rtfused_norm, _silu_routed_e5m2, _v2) to compute intermediate_size from
size(1) and dispatch the _nmajor up kernels.

This lets the host pass w13 without a transposed [E, hidden, 2*inter] replica,
eliminating a full extra weight copy (~10GB/tile at E=256) that previously
forced mem-fraction-static=0.9. Bit-exact vs the K-major path (VNNI identity);
microbench shows only 2-6% per-call cost across cache conditions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…copy)

Refresh patches/sglang_for_multi_arc.patch from origin/dev-bmg so the Docker
build picks up the e5m2 fused-MoE N-major w13 change: fp8.py drops the load-time
_esimd_e5m2_t transpose copy (~10GB/tile), qwen2_moe.py feeds native N-major
w13/shared gate_up, triton.py drops _e5m2_gate_up_transposed.

Regenerated via scripts/update_sglang_patch.sh (git diff v0.5.13 origin/dev-bmg
-- python/); verified it applies cleanly to a pristine v0.5.13 worktree.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

chunk_gated_delta_rule_extend only returned the final recurrent state, so a
mamba track snapshot requested at a position that is not chunk-aligned had
nothing to read.

Add an h_chunk_size argument: when non-zero the kernel also writes h[i], the
state after i full chunks, so the caller can pick the entry covering the
snapshot position. Sequence s owns ceil(L_s/chunk) entries and the sum is
bounded by T_total/chunk + n_seqs, which is what gets allocated; trailing rows
are never indexed. When zero the buffer is empty and the kernel is unchanged.

Pairs with the sglang-side change that threads intermediate_chunk_size through
the torch/triton wrappers.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…pert count

esimd_norm_gemv_q8_0 assumed a single token: x/z were [HV, V] and the WG grid
covered only the output bands. Infer M from the row count and lay the grid out
as M*blocks groups (token = gid / blocks, blk = gid % blocks) so a batched
decode step is one launch. The weight offset within K stays lid*V and is shared
by all tokens, so weights are read once per band and reused, while only the
activation offset carries the extra token*K. Single-token behaviour is the
M == 1 case and is unchanged. This is what lets the fused GGUF MoE path serve
decode batches above M=1.

Also reject n_experts > 512 up front: the topk kernel keeps every expert score
in one simd<fp16, PAD> and the host only instantiates PAD = 256 or 512, so a
larger count silently indexed past the end of that vector.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…n blocks

The final PF_KV_CHUNK iteration covers up to 127 KV rows past seq_len. Their
scores are masked to FP32_MIN so they are numerically inert, but the address
computation still runs for them and indexed block_table_row[] past the blocks
belonging to this request. block_table has exactly ceil(max_seq_len_k /
page_size) columns, so the longest request in the batch fills its row
completely: the read then runs off the end of the row, and for the last request
off the end of the whole tensor. That out-of-bounds device read surfaced as
UR_RESULT_ERROR_DEVICE_LOST.

Clamp the logical block index to the last block the request actually owns. The
loaded values are discarded by the mask either way, so results are unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…stic output)

The softmax running max, the partial sums and the S tiles are all exchanged
between subgroups through SLM. The split named-barrier form
(__esimd_nbarrier_arrive / __esimd_nbarrier) only orders execution: it does not
make the SLM stores visible to the other subgroups, so a subgroup could read
the previous iteration's values. barrier() carries the required memory fence.

The failure matched that mechanism exactly: prompts of at most 128 tokens (a
single KV chunk, no cross-iteration reuse) were always reproducible, while
multi-chunk prompts diverged more often the more chunks they had -- 1/7 runs at
384 tokens, 7/7 at 1024 and 2427 (max_diff 7.0e-03), measured by calling the
kernel directly on fixed tensors. This is also the only kernel in the tree that
uses named barriers; every other one already uses barrier().

After the change all of those are bit-identical (0/7, max_diff exactly 0), and
end-to-end a 27897-token prompt reproduces its output exactly. Cost is +3.3% on
the prefill attention kernel (6.10 -> 6.30 ms at 8192 tokens); decode does not
use this kernel. PF_SPLIT_BARRIER=1 restores the old racy form for A/B testing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

A one-off out-of-tree build wrote to build_opt/, which `build/` did not match,
so 122 MB of stale .so artefacts showed up as untracked files.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…fusions)

Regenerated from sglang v0.5.13..dev-bmg. Beyond refreshing the existing
content this brings in:

- fp8 prefill determinism: narrow-N GEMMs now go to the ESIMD kernel instead of
  torch._scaled_mm. oneDNN picks a split-K schedule for N=32 whose atomic
  reduction order depends on GPU occupancy, so a chunked-prefill tail block
  (M=73, above the old M<=64 ESIMD cap) returned one of two different results
  for byte-identical inputs, and the divergence reached the sampled token.
- GGUF decode determinism: layer 0 has residual=None and therefore skipped the
  fused residual-add/norm/projection path, falling back to the same
  non-reproducible oneDNN GEMV. It now feeds a zero residual and stays fused,
  taking the fused GDN layer count from 78 to 80.
- GDN per-chunk state materialisation for non-aligned extend snapshots, pairing
  with the kernel-side h_chunk_size argument.
- Fused GGUF MoE batch cap raised, with repack chunks sized by bytes.
- An opt-in per-module forward fingerprint trace, off unless its env var is set.

Both paths are now bit-identical across repeated runs: GGUF and fp8 both
reproduce exactly at 9304/27897/83520-token prompts and on the captured BFCL
prompt set.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>